Papers with scientific reasoning
SymPyBench: A Dynamic Benchmark for Scientific Reasoning with Executable Python Code (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing benchmarks do not capture the complexity of structured, step-by-step reasoning essential in physics and related domains. |
| Approach: | They propose a large-scale synthetic benchmark of 15K university-level physics problems . they use structured, step-by-step reasoning and executable Python code to produce the ground-truth solution. |
| Outcome: | The proposed model is based on a set of 15K university-level physics problems with three question types. |
MSEarth: A Multimodal Benchmark for Earth Science Phenomenon Discovery with MLLMs (2026.acl-long)
Copied to clipboard
Xiangyu Zhao, Wanghan Xu, Bo Liu, Yuhao Zhou, Fenghua Ling, Ben Fei, Xiaoyu Yue, Lei Bai, Wenlong Zhang, Xiao-Ming Wu
| Challenge: | Existing datasets often rely on synthetic data or figure-caption pairs, failing to capture the depth and complexity of geoscientific reasoning. |
| Approach: | They propose a multimodal scientific dataset and benchmark curated from open-access publications. |
| Outcome: | MSEarth features over 289K figures with captions enriched by contextual discussions and reasoning from original papers. |
LMR-BENCH: Evaluating LLM Agent’s Ability on Reproducing Language Modeling Research (2025.emnlp-main)
Copied to clipboard
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, Juntong Ni, George Michalopoulos, Yue Zhang, Ziyang Zhang, Mian Zhang, Zhiyu Chen, Xinya Du
| Challenge: | Large language model (LLM) agents have demonstrated remarkable potential in advancing scientific discovery, but their capability in reproducing code from research papers remains underexplored. |
| Approach: | They propose to evaluate LLM agents' ability to reproduce scientific research papers by analyzing code reproduction tasks from 23 research papers published in top-tier NLP venues. |
| Outcome: | The proposed benchmark systematically evaluates the capability of large language model (LLM) agents on code reproduction from Language Modeling Research. |
Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Benchmark (2026.findings-eacl)
Copied to clipboard
Mohammad Khodadad, Ali Shiraee Kasmaee, Mahdi Astaraki, Nicholas Sherck, Hamidreza Mahyar, Soheila Samiee
| Challenge: | a new pipeline for compositional multi-hop reasoning in large language models is being developed . a recent study shows that even state-of-the-art models struggle with compositional reasoning . |
| Approach: | They propose a pipeline that builds benchmarks from proprietary or public data . they use generative reasoning models, chemical named-entity recognition, and external knowledge bases to build knowledge graphs. |
| Outcome: | The proposed pipeline compares state-of-the-art models with and without retrieval augmentation . the pipeline is generalizable with fine-tuning, enabling creation of challenging benchmarks . |
PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers. |
| Approach: | They propose a multi-modal multi-document benchmark for agentic deep research that integrates evidence from multiple documents. |
| Outcome: | Experimental results show that even advanced systems achieve limited scores on PaperScope . paper provides a rigorous benchmark alongside a pipeline for constructing large multi-modal, multi-source deep research datasets. |
From Awareness to Adaptability: Enhancing Tool Utilization for Scientific Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches enhance reasoning through Chain-of-Thought, Program-ofThough, and Tool-Integration. |
| Approach: | They propose a tool-awareness training method that leverages both forward and backward data generation strategies to strengthen the model’s conscious and selective tool utilization in multi-step reasoning tasks. |
| Outcome: | The proposed method improves the model's tool utilization capabilities, including proactivity and execution success rates. |
Monte Carlo Thought Search: Large Language Model Querying for Complex Scientific Reasoning in Catalyst Design (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a goal-driven combinatorial search using large language models has not been explored in detail. |
| Approach: | They propose a Monte Carlo Tree Search-based approach that improves beyond state-of-the-art chain-of thought prompting variants to augment scientific reasoning. |
| Outcome: | The proposed approach improves over the best baseline by 25.8% and can augment scientist’s reasoning and discovery process with novel insights. |
WildSci: Advancing Scientific Reasoning from In-the-Wild Literature (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in large language model reasoning focus on mathematics and coding domains, but scientific reasoning remains limited in other domains due to limited dataset coverage. |
| Approach: | They propose a framework for sustainable scientific reasoning QA generation by synthesizing a new dataset of domain-specific science questions from peer-reviewed literature. |
| Outcome: | The proposed framework and dataset enable scalable and sustainable research in scientific reasoning. |
Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science (2026.findings-acl)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) outperform existing benchmarks in both natural language and coding domains. |
| Approach: | They propose a scalable benchmark that integrates vision and language modalities to address this gap by eliminating textual shortcuts. |
| Outcome: | The new benchmark outperforms existing benchmarks in both natural language and coding domains. |
Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla P Gomes, Bart Selman, Qingsong Wen
| Challenge: | Current scientific reasoning models struggle with generalization across domains and fall short of multimodal perception. |
| Approach: | They propose to use multimodal large language models to integrate text, images, and other modalities to enhance scientific reasoning. |
| Outcome: | The proposed models can integrate text, images, and other modalities and improve reasoning across disciplines. |